EvalUMAP: Towards Comparative Evaluation in User Modeling, Adaptation and Personalization

The 1 International EvalUMAP Workshop brings together researchers from every facet of the User Modelling, Adaptation and Personalization community to explore new approaches to comparative evaluation. This area of research is inherently complex, not only because of its focus on the user, but also because the domains in which it is applied tend to be highly dynamic and diverse. To date this has presented significant barriers to how research outcomes can be compared. This workshop seeks to solicit input from the community to facilitate the shared development of evaluation tasks or competitions. The ADAPT Centre (www.adaptcentre.ie) is committed to supporting this community-wide activity as it will, hopefully, strengthen the research of all involved. 1. PREFACE Research in the areas of User Modelling, Adaptation and Personalization faces a number of significant scientific challenges. One of the most significant of these challenges is the issue of comparative evaluation. It has always been difficult to rigorously compare different approaches to personalization, as the function of the resulting systems is, by their nature, heavily influenced by the behavior of the users involved in trialing the systems. To-date this topic has received relatively little attention. Developing comparative evaluations in this space would be a huge advancement as it would enable shared comparison across research, which to-date has been very limited. Taking inspiration from communities such as Information Retrieval and Machine Translation, the first EvalUMAP Workshop seeks to propose and design one or more shared tasks to support the comparative evaluation of approaches to User Modelling, Adaptation and Personalization. The workshop has successfully solicited presentations from key practitioners in the field on innovative approaches to evaluating such systems and it provides a forum to start scoping and designing tasks for subsequent years. The resulting shared task(s) will be accompanied by appropriate models, content, metadata, user behaviors, etc., and can be used to comprehensively compare how different approaches and systems perform. In addition, a number of metrics and observations will be outlined, that participants would be expected to perform in order to facilitate comparison. The following topics are addressed by this workshop.  Understanding UMAP evaluation  Defining tasks and scenarios for evaluation purposes  Identification of potential corpora for shared tasks  Interesting target tasks and explanations of their importance  Critiques or comparisons of existing evaluation metrics and methods  How we can combine existing evaluation metrics and methods  Improving on previously suggested metrics and methods  Reducing the cost of evaluation  Proposal of new evaluation metrics and methods  Technical challenges associated with design and implementation  Privacy, Ethics and security issues  Legal and ethical issues EvalUMAP is designed to be an interactive workshop structured to encourage group discussion and active collaboration among attendees. The workshop will feature a keynote talk, lightning round presentation session for position papers, multiple (parallel) breakout sessions, and a final discussion session to wrap up the event. The planned outcome of the EvalUMAP Workshop 2016 will be a roadmap to develop initial shared task(s) that will be published well in advance of UMAP 2017, giving an opportunity for participants to test and tune their systems and complete the task in order for comparative results and associated publications to be prepared for and presented at the EvalUMAP Workshop in 2017. We envision that EvalUMAP 2017 will be the starting point for an annual comparative evaluation challenge at future UMAP conferences. 2. PROGRAMME COMMITTEE The programmee committee consists of the following members:  Paul De Bra, Eindhoven University of Technology, The Netherlands  Iván Cantador, Universidad Autónoma de Madrid, Spain  David Chin, University of Hawaii, USA  Eelco Herder, L3S Research Center, Hannover, Germany  Geert-Jan Houben, Delft University of Technology, The Netherlands  Judy Kay, University of Sydney, Australia  Tsvi Kuflik, The University of Haifa, Israel  Alexandros Paramythis, Contexity, Switzerland  Francesco Ricci, University of Rome, Italy  Alan Said, Recorded Future, Sweden  Vincent Wade, Trinity College Dublin, Ireland  Stephan Weibelzahl, Private University of Applied Sciences Göttingen, Germany